Skip to content

H1: add validated feed primitive status - #41

Closed
Pigbibi wants to merge 2 commits into
mainfrom
feat/pert-upstream-feed-primitive-status-h1
Closed

H1: add validated feed primitive status#41
Pigbibi wants to merge 2 commits into
mainfrom
feat/pert-upstream-feed-primitive-status-h1

Conversation

@Pigbibi

@Pigbibi Pigbibi commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Scope

  • Add immutable validated primitive row snapshots and a typed pert.feed_primitives.v1 status contract.
  • Bind per-feed row counts/digests and aggregate row digest from the same one-pass row snapshot.
  • Canonical serializer/parser rejects unknown, duplicate, noncanonical, malformed, unsafe integer, and state-invariant violations.
  • Integrate the existing producer to emit the canonical status while preserving the existing RSS/Atom row CSV fields and partial/debug flow.

Contract

  • H1 public APIs accept primitive mappings/rows only; no raw XML or network response.
  • Producer raw XML remains behind the merged H0 defusedxml boundary.
  • Supported outcomes are only producer-observed accepted, failed, and quarantined; stale/missing are not invented.
  • Zero-row feeds are quarantined and never publication-eligible.
  • Feed-record input ordering is normalized; row order remains producer-semantic.

Verification

  • uv sync --locked --extra test passed
  • uv run pytest -q — 135 passed
  • python3 -m compileall -q src scripts tests passed
  • git diff --check passed
  • ruff unavailable in the environment

No workflow, publish, QAR, weekly artifact, permission, or automation changes. W0-B remains deferred until this PR merges and post-merge CI succeeds.

Co-Authored-By: Codex <noreply@openai.com>
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🤖 Codex PR Review

🚫 Merge blocked: 2 serious issue(s) found in high-risk files

⚖️ Codex Review Arbitration

🚫 block: rss_source_fetch.py proves the second finding remains valid: zero-entry feeds are now emitted as {"state": "quarantined", "rows": [], "error_code": "zero_entries"}, but fetch_rss_sources() still raises RuntimeError("all configured RSS/Atom feeds failed") whenever no feed is accepted. Therefore a run where every feed fetches/parses successfully yet yields zero entries now hard-fails, whereas the prior implementation treated those feeds as successful with item_count == 0. The first finding is also still supported by the new contract code: in feed_primitives.py, _snapshot_feed() allows state == "quarantined" whenever error_code is present, even if rows is non-empty; _feed_wire() then serializes those rows into nonzero accepted_row_count and a non-empty row_digest; and _validate_wire() only checks that quarantined feeds have an error_code, not that their row snapshot is empty. No prior finding conflicts with these conclusions; the earlier safe-integer blocker was addressed separately by MAX_SAFE_JSON_INTEGER and _safe_int().

🚫 Blocking Issues

These issues must be fixed before this PR can be merged:

1. 🟠 [HIGH] Logic in src/political_event_tracking_research/feed_primitives.py

build_status() accepts state == "quarantined" records as long as error_code is present, even when rows is non-empty. _validate_wire() has the same gap for accepted_row_count/row_digest. That means callers can produce a supposedly valid status where quarantined feeds contribute accepted rows and digests, which violates the PR contract that quarantined feeds are zero-row and not publication-eligible. (line 148)

Suggestion: Reject quarantined feeds unless their row snapshot is empty. In _snapshot_feed() require not rows, and in _validate_wire() require accepted_row_count == 0, rejected_row_count == 0, and the empty-row digest for quarantined feeds.

2. 🟠 [HIGH] Logic in src/political_event_tracking_research/rss_source_fetch.py

The producer now raises RuntimeError("all configured RSS/Atom feeds failed") whenever no feed is in the accepted state. Because zero-entry feeds are now mapped to quarantined, a run where every feed fetched and parsed successfully but happened to have zero entries will abort as a total failure. That is a behavior regression from the previous implementation, which treated those feeds as successful with item_count == 0. (line 220)

Suggestion: Only raise the "all failed" error when every feed is actually failed, or add a separate no-data/quarantined path that writes the status and returns an empty CSV without turning the run into a hard failure.

ℹ️ Other Findings

1. 🟡 [MEDIUM] Logic in src/political_event_tracking_research/feed_primitives.py

PrimitiveRow.from_mapping() treats published_at as any non-empty string. Since build_status() is now a public validated-row API, this allows non-canonical or even syntactically invalid timestamps to enter the canonical wire format. Those raw strings are then used for row ordering and digests, so semantically identical rows can hash differently and sort incorrectly across producers. (line 95)

Suggestion: Validate published_at against a single canonical wire format (for example exact UTC RFC3339/ISO-8601 YYYY-MM-DDTHH:MM:SSZ) and reject any other representation before accepting the row.


Review by Codex PR Review bot • PR

Co-Authored-By: Codex <noreply@openai.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3b0e66553

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

item_count=0,
error=f"{type(exc).__name__}: {exc}",
)
error_code = exc.code if isinstance(exc, (FeedXmlError, PrimitiveStatusError)) else "fetch_failed"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve FeedXmlError handling under continue mode

When parse_feed_items raises FeedXmlError for invalid or oversized XML while --continue-on-feed-error is set, this line enters the isinstance branch and reads exc.code, but FeedXmlError only carries its string message and has no code attribute. The exception handler then raises AttributeError before appending a failed record or continuing, so one malformed feed can still abort the RSS workflow.

Useful? React with 👍 / 👎.

output_path = Path(path)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n", encoding="utf-8")
output_path.write_bytes(payload + b"\n")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Emit status bytes that the canonical parser accepts

The new status parser enforces canonical bytes by checking serialize_status(parsed) == wire, so the trailing newline written here makes every producer status file fail parse_status_bytes/status_for_rows with status_noncanonical when read back as bytes. If these files are meant to be the canonical contract output, write exactly payload or make the parser explicitly allow the file terminator.

Useful? React with 👍 / 👎.

Comment on lines +224 to +225
if any(type(snapshot[key]) is not int or snapshot[key] < 0 for key in integer_keys):
raise _fail("status_counter_invalid")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject unsafe counter values

These checks only require counters to be nonnegative Python ints, so a crafted status with matching top-level and per-feed accepted_row_count values such as 9007199254740993 is accepted and re-serialized as canonical even though that value is not safe for JSON consumers that preserve integers only up to 2^53. Since this parser is the boundary for the canonical status contract, it should reject counters outside the intended safe range instead of blessing them.

Useful? React with 👍 / 👎.

Comment on lines +144 to +145
if state == "quarantined" and not error_code:
raise _fail("feed_state_invalid")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep quarantined feeds from contributing accepted rows

A caller can pass a quarantined feed with an error_code and non-empty rows, and this invariant accepts it; _feed_wire then reports those rows as accepted_row_count and includes them in the aggregate row digest even though the feed is not publication-eligible. If quarantined rows are meant to be withheld or rejected, this lets invalid primitive records be blessed by the public status builder.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant